Signal buffering in random networks of spiking neurons: 
microscopic vs. macroscopic phenomena 
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In randomly connected networks of pulse-coupled elements a time-dependent input signal can be 
buffered over a short time. We studied the signal buffering properties in simulated networks as a 
function of the networks state, characterized by both the Lyapunov exponent of the microscopic 
dynamics and the macroscopic activity derived from mean-field theory. If all network elements 
receive the same signal, signal buffering over delays comparable to the intrinsic time constant of the 
network elements can be explained by macroscopic properties and works best at the phase transition 
to chaos. However, if only 20 percent of the network units receive a common time-dependent signal, 
signal buffering properties improve and can no longer be attributed to the macroscopic dynamics. 

PACS numbers: 87.18.Sn, 87.10.+e, 82.40.Bj 



INTRODUCTION 

Neurons in the brain are connected with each other and 
send short electrical pulses (action potentials or spikes) 
along those connections. Despite the fact that there are 
correlations between the type of connections and the type 
of neurons , it is fair to say that neurons fall essentially 
into two classes, excitatory and inhibitory, and that the 
connectivity in a local population of several thousand 
cortical neurons is close to random. Networks with fixed 
random connectivity can, in principle, contain loops of 
varying size, which could sustain the flow of transient 
information signals over times that are long compared 
to the intrinsic time constants of the network elements, 
i.e., the neurons. In neuroscience and related fields, el- 
ementary considerations on information flow in random 
networks have inspired ideas as diverse as synfire chain 
activity Q , reverberations Q , liquid computing Q , echo 
state machines or computing at the edge of chaos 
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On the other hand, random networks have also been 
studied intensively by the phy_sics community, in the con- 
formal neural networks 
and limiting cases have been 
identified for which exact solutions are known. In partic- 
ular, in the limit of asymmetric networks with low con- 
nectivity, mean- field dynamics becomes exact 01- More 
recently these approaches have been extended to the case 
of random networks of spiking neurons in continuous time 

In this paper, we will compare simulations of a random 
network of excitatory and inhibitory neurons with the 
mean-field solutions valid in the low-connectivity limit 
and evaluate the performance of such networks on a sim- 
ple information buffering task that can be seen as a mini- 
mal and necessary requirement for more complex compu- 
tational tasks which a neural network might have to 
solve. More precisely, the task consists in reconstructing 



a time-dependent input I it') by reading out the activ- 
ity of the network at a later time t = t' + A. We will 
see that performance in the information buffering task is 
best at the phase transition that is marked by a rapid 
increase in both the macroscopic activity variable and 
the Lyapunov exponent characterizing the microscopic 
state indicating transition to chaos. Surprisingly, if the 
same time-dependent input I(t) is shared by all neurons 
in the network, an information readout based only on the 
macroscopic variable is as good as a readout that is based 
on the output pulses of all N neurons. However, if the 
input is only given to a small group of neurons a detailed 
readout conveys more information than a macroscopic 
one suggesting that loops in the network connectivity 
might indeed play a role. 



MODEL 

We consider a network of N leaky integrate-and-firc 
units (neurons) with fixed random connectivity. 80 per- 
cent of the neurons are taken as excitatory and the re- 
maining 20 percent inhibitory. Independent of the net- 
work size (N — 200,400,800), each neuron i in our 
simulation receives input from Ce = 40 excitatory and 
Cj = 10 inhibitory units (presynaptic neurons), which 
are chosen at random amongst the N — 1 other neurons 
in the network. The ensemble of neurons that are presy- 
naptic to neuron i is denoted by Mi and the efficacy tuy 
of a connection from a presynaptic neuron j € Mi to a 
postsynaptic neuron i is Wij — we if j is excitatory, and 
Wij = Wj if j is inhibitory. 

Each neuron is described by a linear equation com- 
bined with a threshold. In the subthreshold regime the 
membrane potential follows the differential equation 



l (t) = -u l (t)+ir tw +ir t (t) 
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where r m = 20ms is the effective membrane time con- 
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stant and If xt external input. The recurrent input If etw 
neuron i receives from the network is 

Ji(t)=r ra £ £ (2) 

jeMi k 

where tj is the time neuron j fires its kth spike and 
D = 1ms is a short transmission delay. A spike from 
an excitatory (inhibitory) neuron j € Mi causes a jump 
in the membrane potential of neuron i of we = 0.6mV 
(wi = — 3.6mV). If the membrane potential m reaches a 
threshold # = lOmV, a spike of neuron i is recorded and 
its membrane potential is reset to m — 0. Integration 
restarts after an absolute refractory period of r rp = 2ms. 

The external input If xt can be separated into two com- 
ponents. First, we inject a time dependent test signal 
Ii g (t) which is generated as follows: the total simulation 
time is broken into segments of duration T sg = 10ms. 
During each segment of length T sg the input is kept con- 
stant. At the transition to the next segment, a new value 
of P s (t) is drawn from a uniform distribution over the 
interval [—0.25,0.25], i.e. the signal distribution has a 
standard deviation of a sg = ^= = 0.144. By construc- 
tion the signal at time t provides no information about 
the signal at t — T for T > T sg . More precisely, the auto- 
correlation A(s) of the signal has a triangular shape and 
is strictly zero for \s\ > T sg . 

Second, the network of N neurons is considered as part 
of a larger brain structure. To mimic input from excita- 
tory 'background' neurons that are not modelled explic- 
itly, we assume stochastic spike arrival described by a 
Poisson process of total rate ^ exc . For the sake of simplic- 
ity, we assume that the efficacy of background spikes 
is identical to that of the recurrent connections within the 
network. We approximate the background input J back s 
by a Gaussian white noise with mean ^t back s = uj E i/ cxc T m 
and standard deviation cr backg = LdE\/v cxc T mi where f eX c 
is the background spike arrival rate. This approximation 
is valid under the assumption that a neuron receives a 
large number of presynaptic contributions per unit time, 
each generating a change in the membrane potential that 
is relatively small compared to the firing threshold. To 
simplify notation we set 7 = Ci/Ce and g = —loj/lje- 
The values in our simulations are 7 = 0.25 and 5 = 6. 

CHARACTERIZING THE NETWORK STATE 

If we replace the signal I ss by Gaussian white noise of 
zero mean and standard deviation a mean-field anal- 
ysis of the random network defined above can be per- 
formed following |l2^ . The macroscopic variable describ- 
ing the activity of the network is the population rate vq. 
In a stationary state of asynchronous neuronal activity, 
the population rate depends on the mean 

^0 = ^BT m (^cxc + C E ^o(1 - .97)) (3) 



and variance 

erg = ^IttoOcxc + C E fo(l + 9 2 l)) + ofg (4) 
of the total input / nctw + / oxt via the equation 

i) — LLC) 

— = r rp + 2tt / due u2 / dve~ v2 (5) 

The mean-field result is valid in the low connectivity limit 
[l2|. Combining Eqs. © - ©7 we obtain a self-consistent 
solution of the population firing rate vq as a function 
of the external Poisson drive v cxc . The population rate 
shows a marked increase near v cxc = 420Hz as shown 
in Fig. ^ which is in the vicinity of a first-order phase 
transition predicted by the mean-field theory. If we de- 
crease the amplitude of the input signal cr sg — > the 
phase transition becomes more pronounced and a regime 
of coexistence of several solutions appears (inset). 
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FIG. 1: Dashed line: Fixed points of the population activ- 
ity Vo as a function of the Poisson background drive resulting 
from Eqs. 13151 . Note the sharp transition at v eyic = 420Hz. 
The network switches from an almost quiescent state to a 
state of sustained activity. Dotted line: largest Lyapunov ex- 
ponent as a function of the external drive v cxc in a network 
of 800 neurons. For f cxc < 420Hz the population rate vq is 
low ('quiescent state') and the largest Lyapunov exponent is 
negative. For a stronger drive, the exponent switches to a pos- 
itive value, reflecting the chaotic behaviour of the membrane 
potential trajectories. Solid line: Signal reconstruction error 
for a delay of 20 ms in a network of 800 neurons. The error 
is minimal near the transition from the quiescent to a chaotic 
regime. Inset: Fixed points of the population rate (Eq. |SJ in 
absence of test signal (solid line), and with increasing signal 
variance. The system exhibit a first-order phase transition if 
the signal is weak. 

The marked increase in the macroscopic variable Vq 
is accompanied by a transition of the largest Lyapunov 
exponent from negative to positive values which indi- 
cates that the microscopic dynamics becomes chaotic. 
The largest Lyapunov exponent is defined by A p = 
limt^cx, lnjj& where e(t) = X^ili^W _ u i( f )) 2 is tri e 
difference between a reference trajectory of the N vari- 
ables «i(t), ...iijv(t) and test trajectory u'i(t), ...Ujv(i)' 
with slightly different initial conditions. Using standard 
numerical techniques 13], the largest Lyapunov exponent 



has been estimated from a large number of simulations 
of a test network and a reference network with identi- 
cal connectivity. Both networks received identical input 
(same realization of the Poisson input and signal input) 
and the test trajectory was regularly reset close to the 
reference trajectory. Moreover, we confirmed that not 
only for background spike input but also for appropri- 
ately chosen constant input (i.e. /i back s > 0, <j back s = 0), 
the fixed random connectivity is sufficient to generate ir- 
regular asynchronous spiking activity with a positive 
largest Lyapunov exponent (data not shown). 



INFORMATION BUFFERING 

After having characterized the macroscopic and micro- 
scopic state of the network, we asked how performance 
in an information buffering task, inspired by concepts of 
liquid computing Q and echo state machines would 
depend on the network state. We considered a linear 
readout unit with dynamics dy/dt = — ((y — cxo)/t s ) + 
YliLi a i Ylk ^ — rf) where the sums run over all firing 
times t\ of all neurons in the network. t s = 5ms is a 
short synaptic time constant. The N + 1 free parameters 
oti (0 < i < N) are chosen so as to minimize the signal 
reconstruction error E — {[y(t) — P s (t — A)] 2 )/of g . Pa- 
rameters were optimized using a first simulation (learning 
set) lasting 100 seconds (lOO'OOO time steps of simula- 
tion) and were kept fixed afterwards. The performance 
measurements reported in this paper are then evaluated 
on a second simulation of 100 seconds (test set). Simula- 
tion results were obtained using the simulation software 

nestHU. 

The same time-dependent signal I ss (t) was injected 
into all neurons in the network and the performance 
evaluated in terms of the signal reconstruction error. 
The performance depends on the delay A of information 
buffering which has to be compared with the membrane 
time constant (r m — 20ms) and the autocorrelation of 
the input T sg = 10ms. Overall the signal reconstruction 
error is relatively high. As expected, the signal recon- 
struction error increases if we increase the desired buffer 
duration from A = 10ms to A = 15ms or A = 20ms 
(Fig. |2K)- At the same time, the optimal background 
firing rate v cxc to achieve minimal signal reconstruction 
error shifts towards lower values and is for A = 20ms 
very close to the transition between regular and chaotic 
microscopic dynamics as shown in Fig. ^ This result is 
consistent with the idea of computing at the edge of chaos 
in cellular automata or threshold gates [|| . Also, sim- 
ilar to the results in discrete-time spin networks |lj] , the 
information buffering performance does not significantly 
depend on the number ./V of neurons in the network; cf. 
Fig. 03. Differences are within the statistical variations 
caused by overfitting on finite data samples. 
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FIG. 2: A. Signal reconstruction error £ as a function of 
the background firing rate in a network of 800 neurons for 
three different information buffer delays: A = 10ms (solid), 
A = 15ms (dashed), A = 20ms (dotted). For sufficiently 
long delays, optimal performance is located near the transi- 
tion between the quiescent and the chaotic state; cf Fig Q 
Deeper in the chaotic phase the error goes back to the chance 
level whereas in the almost quiescent regime we can see the 
effects of overfitting (E > 1), because the number of action 
potentials is insufficient to build an accurate model of the 
past events. B. Comparison of the errors for different net- 
work sizes: N = 200, 400, 800 neurons (resp. dotted, dashed 
and solid lines), for a delay A = 20ms. The solid line with 
filled circles corresponds to the macroscopic readout of the 
network of N = 800 neurons. The location of minimal error is 
independent of the number of neurons and coincides with the 
phase transition. The vertical shift of the error curves is not 
significant but due to overfitting because of limited amount 
of data. Vertical bars indicate the mean difference between 
errors on the data used for parameter optimisation (training 
set) and that on an independent test set for N = 800 (left 
bar), N — 400 (2nd bar), N = 200 (3rd bar) and macroscopic 
readout (right bar). A representative curve of errors on the 
training set for N = 800 neurons is shown by the dot-dashed 
line. 



MICROSCOPIC VS. MACROSCOPIC 
PROPERTIES 

Given that networks states have been classified suc- 
cessfully by macroscopic mean-field equations E3j we 
wondered whether the performance in the above infor- 
mation buffering task can be completely understood in 
terms of macroscopic quantities. To answer this ques- 
tion, we compared the performance using the previous 
readout unit y (i.e., a 'microscopic' readout with N + 1 
free parameters, one per neuron plus an offset) with that 
by a simplified readout y with two free parameters a and 
only, dy/dt = -((y - 0)/t.) + a^ti Efc *(* " 
i.e., a readout that uses only the macroscopic population 
activity. Surprisingly, for a stimulation paradigm where 
all neurons receive the same time-dependent signal J sg (t) 
the macroscopic readout performs as well as the micro- 
scopic one. In other words, connectivity loops between 
specific subsets of neurons where signals could circulate 
for some time seem not to play a role in information 
buffering. This suggests that, for signals of sufficiently 
small amplitude, the information buffering capacity is 
directly related to the macroscopic linear response ker- 
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nel k of the network activity, that can, in principle, be 
calculated from the linearized mean-field equations of the 
population rate, i.e., Av(t) = J k(s) P s (t — s)ds; cf. [l2j| . 
The time constant of the kernel, and hence information 
buffering delays, become large in the vicinity of a phase 
transition. 

We hypothesized that signal transmission loops in our 
randomly connected network could manifest themselves 
more easily if only a small subset of neurons received 
the input signal. We therefore selected 20 percent of 
neurons at random (group G\) and injected an identical 
time dependent signal Ij S (t) into all neurons j € G\. The 
remaining 80 percent of neurons (group G2) received no 
signal. In such a network consisting of two groups, signal 
buffering performance is indeed significantly better than 
in a single homogeneous group (Fig. [3Jl. 

On a macroscopic scale, a network consisting of two 
groups G\ and G2 can be described by two macroscopic 
variables, i.e., the population activities in groups G\ and 
G2 . In order to evaluate the information contained in the 
macroscopic population rates, we used a linear readout 
unit ?/2 with three free parameters /3o, Pi and fc, char- 
acterized by the differential equation di/2/dt — —((2/2 — 

fa)lr.)+fh £ <6Gl E <eGa E fc 

and proceeded as before. As we can see from Fig. |3 
a readout based on the macroscopic activity of the two 
groups performs significantly worse than the microscopic 
readout. This suggests, that for the case when only a 
small subset of units in a random network receive an 
input, signal transmission loops, and hence microscopic 
neuronal dynamics, indeed play a role in short-term in- 
formation buffering. 
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FIG. 3: The input is injected to 20% of the neurons only. 
A macroscopic readout assuming a single population (dotted 
line) performs well near the phase transition. However deeper 
in the chaotic phase it is outperformed by the microscopic 
readout (solid line). A macroscopic readout based on a two- 
population assumption (dashed line) explains only part of the 
increased performance. The signal buffering delay for this 
figure is A = 20ms. 



DISCUSSION 

Mean-field methods neglect correlations in the input. 
In random networks mean-field theory becomes asymp- 
totically correct only in the low-connectivity limit where 
the probability of closed loops tends to zero ^(J- How- 
ever, it is exactly these loops which could give the net- 
work the power to buffer information for times signifi- 
cantly longer than the intrinsic time constants of the net- 
work elements. Our network is formally not in the low- 
connectivity limit since the number of neurons N = 800 
is small. Nevertheless, we found that mean- field results 
can qualitatively predict the rough location of the phase 
transition of the macroscopic population rate. Moreover, 
if the input signal is shared between all neurons, a macro- 
scopic readout is sufficient to explain the network per- 
formance in an information buffering task. Microscopic 
properties do, however, play a role if the input is only 
given to a subset of the network units suggesting that in 
this case ultra-short term information buffering in con- 
nectivity loops is indeed possible. In additional simula- 
tions we checked that the maximum delay A for which 
signal reconstruction is feasible is only in the range of 
20-50ms and hence not significantly different from the in- 
trinsic neuronal time constants. This suggests that, with- 
out slow processes such as synaptic plasticity or neuronal 
adaptation a purely random network of spiking neurons 
is not suitable as an information buffer beyond tens of 
milliseconds. 
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